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Computer vision, especially color image analysis and understanding, much to offer in the area of the automation of Space Station 
such as construction, satellite servicing, rendezvous and proximity operations, inspection, experiment monitoring, data management and 
training. Knowledge- based techniques improve the performance of vision algorithms for unstructured environments because of their ability to 
deal with imprecise a priori information or inaccurately estimated feature data and still produce useful results. Conventional techniques using 
statistical and purely model-based approaches lack flexibility in dealing with the variabilities anticipated in the unstructured viewing environment 
of space. 


Algorithms developed under NASA sponsorship for Space Station applications to demonstrate the value of a hypothesized architecture 
for a Video Image Processor (VIP) are presented. Approaches to the enhancement of the performance of these algorithms with knowledge- 
based techniques and the potential for deployment of highly-parailcl multi-processor systems few these algorithms are discussed. 

1.0 Introduction 

* 

A major consideration in the design and deployment of the NASA Space Station is the definition of automation techniques which will 
guarantee the timely and reliable performance of the Space Station's missions. During specification of the initial design of the Space Station, 
NASA has identified three criteria for the justification of the development of an automation technique: 1 


1. The automation capability should be of substantial value toward the objective of accomplishing Space Station functions, such as user 

experiment monitoring, user production activities and satellite servicing in a timely and reliable manner. 

2. The safety of the crew must not be compromised. 

3. The Space Station should operate autonomously with as little support from ground-based facilities as possible. 

A Video Image Processor will be a very valuable automation tool on-board the Space Station for several reasons: Image processing, 
specifically the identification of the objects seen in the image and the formulation of a 3-dimensional model of a scene, is a pre-requisite 
capability for the development of autonomous robots. These autonomous robots could perform many of the mundane tasks such as experiment 
monitoring and proximity operations that at the present time require crew member supervision. Image processing is also a pre-requisite 
capability for the task of bandwidth reduction which will be necessary for the Space Station because of limited on-board storage and the 
restraints of secure channel downlinks from the Space Station to ground-based facilities. For semi-autonomous implementations, image 
processing is employed to execute repetitive tasks such as color image enhancement/restoration or operator cueing, and an operator is required 
only for verification confirmation of the actions of the algorithms. A Video Image Processor can perform each of the preceding tasks, thus 
increasing the efficiency of crew members of the Space Station. 

A Video Image Processor is a dedicated processing unit for image data that is modularly extendable and is to be built from commercially 
available components. The VIP architecture was defined conceptually under contract to NASA by Honeywell Systems and Research Center on 
the basis of several criteria, which are: maintainability, extensibility, programmability, physical aspects of deployment and the performance 
specifications defined by current Space Station applications^. The candidate architectures for the VIP were quantitatively evaluated with 
architecture analysis tools to obtain a high degree of confidence in achieving the desired functionality. To do this a set of example image 
processing algorithms had to be specified and their performance evaluated for imagery acquired during previous Space Shuttle missions to 
simulate the algorithms' behavior under realistic conditions. In this way, the processing requirements of the algorithms could be estimated for 
the unique set of environmental, lighting and imaging constraints found in space. 

The goal of the selection process was to develop an algorithm suite that would benefit a sufficiently large number of space station tasks. 
The various space station tasks that benefit from an image processing capability can be classified into eight generic categories.^ 

• Construction 

• Satellite servicing 

• Rendezvous and proximity operations 

• Inspection 

• Payload delivery and retrieval 

• Experiment monitoring 

• Data management and communication 

• Training 

In order for the VIP to assist in the automation of these tasks, it should have a substantial array of image processing algorithms that it 
can apply in accordance with the changing demands of the application. These image processing algorithms can be grouped into six major 
families for the space station scenario: 

• Color image enhancement 

• Tracking 

• Surveillance 

• Identification 
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• Proximity operations 

• Bandwidth reduction 

are sufficiently mature fee the design and build of a prototype system. 
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Station tasks. 

This cm be tttributedtotwofsciors. Fust. 

conditions. Imaging systems that employ ■ ^ thennal and electrical noise in the imaging system hardware. When an analog 
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response time of the digitization amplifiers. 

Even if the image's quality i. very good. 

(CCDh .he perfonoMceofdK image pMc^ni S^iST^^uL™^^ 

algorithms. The reason that nowe ^^^Jfb«2ue of variations in surface smoothness, and therefore images of these 

itvmagnitudc wuPb^bSow the detection thresholds of the image processing algorithms. 

Generic Tracking algorithms are also a pre-requisite capability for illofthe Space Station tasks considered. This can be explained as 
follows: Scene interpretation algorithms can be decomposed into four stages. 

t. Segmentation 

2. Feature Detection 

3. iconic (Pixei-Based)-to-Symbohc Feature Mapping 

4. Classification 

The algorithms that together form theTracUng^ncdonMe^^MlertW^hefiraATO^a^deMibedabOTe^ 

Al^^ing of d* imagetalo nsgionjof inreres. Md ««neou S hMkgroend is .he ininal 
ph£e of all image 5 processing jdgorithms that are designed to obtain symbolic information from raw image data. 

cigh . 

for the Tracking function, with the exception the temporal silhouette matching algorithm required for Tracking. 

Knowledce-based techniques are a means of employing the efficient symbolic pattern matching and high-level reasoning capabilities of 
arufici J^^^Ugowe ^ inwge interpretation applications. Knowledge-based techniques for region labeling can roleratelarge^ in featine 
, , r ... rrunaninafni n»qnlts Thev can be designed in stages because of their modular rule database, as new contexts are discovered 

St^,r?r U »M^nffSms 8 !he CTSicm is reconfigured by the definition or modificalion of a few rules. Knowledge-based systems arc very 
^c“r«h o^Kn large symboiic darabases on fire basis of relariooal and conie*ual eonsinunrs on ihc 

data. 

Due to .he unpredictable naiure of imagery obtained in space . especiafiy dining 

mbots ^ other f act ors unique to a space environment such as rapid diurnal changes while a vehicle is in orbit, krowieage oasea 

logical statements of the form ’If .... then...” and a control system that supervises rule acuvauon. 

The svstem developed by Nagao and Matsuyama 5 uses a knowledge base representing relational contextual and geometric constants 
for the task ofre^onSmE for multi- spectral imagery obtained from low-flying aircraft. The region boundaries are detected by a variety of 
low-level image segmentation algorithms and the resultant information is archived on a blackboard shared by each of the experts of system. 
F^h exii^s^SSd^ locating a specific kind of object or region. They devised an approach for the reliable classification ofvegemuonal 
Each expert is optimized tor local g pec j e distinct spectral bands to discriminate the vegetation regions from the non- 

74^4 retabk '^“ ficauo " of tascs “ d “* m c “ ec5,cd “ b “ 

scenes where other classical approaches normally fail. 

Ohta 9 developed a hierarchical region labeling scheme for color images of urban scenes. The approach is hicrarchKal because an initial 
unta oeveiopea a nierarcmcai h . ■£. data-directed segmentation is carried out. The plan image is defined by a region-based 

scenes using only 57 rules. 

Howrver verv little work has been done in the area of the application of knowledge-based techniques for tracking or motion 
understanding This Is primarily due to the unconstrained nature of the problem. It is exceedingly difficult to specify an !”P e ^ sl *" 1 .*[“* JjJJ 
characterize the dvnarm^behavior of arbitrary objects as they translate and rotate in three dimensions. Work has been done on dynamic 

rS employing an intelligent system for reasoning about the three-dimensional structure of a 
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stationary environment 10 . This approach uses path planning techniques for the notification and avoidance of obstaclodetectcd by sonar 
sensors. The technique was successfully demonstrated for the task of obstacle avoidance while navigating an Ml 13 autonomous veh ic ! c ’ bui1 ! 
by FMC Corporation for the DARPA ALV program, through a maze of obstacles. However, no procedure is specified for the detection and 
interpretation of sensor data that results from moving objects. 

Honeywell is carrying out research on the utilization of knowledge-based techniques for the discrimination of moving objects from 
stationary objects in imagery obtained from a moving autonomous vehicle's camera. 11 This approach, which we have labeled Dynamic 
Reasoning from Integrated Visual Evidence (DRIVE), identifies visual cues from a sequence of images that defines a global dynamic reference 
model. Object recognition, world knowledge and the accumulation of evidence are used to disambiguate the situation and refine the global 

reference model 

The design principles for the identification of the optimal architecture for the Video Image Processor were reviewed in this section. The 
figures of merit for the design were stated and the approach utilized to validate the design for space imagery were summanzedThe raoonale for 
selecting the three image processing algorithms which were studied was explained. The next section discusses the technical details of the three 
image mocessing algSthmsand specifies approaches for executing the same algorithms with knowledge-basedtrchniques. Secuon3 
summarizes approaches that may be pursued for the development of image interpretation systems that employ knowledge-based t«hniqtws. 
TtefomAwuon describes experimental results obtained for the algorithm validation task for the Video Image Processor. Section 5 discusses 
a few important problems that are yet to be solved and that can produce significant increases in algorithm efficiency and reliability for image 
processing in a space scenario. 


2.0 VIP Algorithms 

The details of the implementations of the three image processing functions chosen for the establishment of space processing 
requirements for the VIP are discussed in this section. Special attention is paid to citing how the performance of each algorithm is enhanced 
with knowledge-based techniques. 


2 1 Color ImaEC F.nh ancement -The following is a description of three of the algorithms that were evaluated for Color Image Enhancement 
Each algorithm is designed to restore a particular feature of the color images, e.g., dynamic range, sharpness, etc. It is therefore conceivable 
that an actual implementation may use combinations of these algorithms to produce imagery with specific color characteristics. 


2 1 I rnlnr Image Balanced Histogram Equalization -Color image balanced histogram equalization enhances image contrast and increases 
image dyn amic ran ee The a 1 ^rit h m opera te s wi ih the same fundamental principle that monochrome image histogram equalization employs, 
namely that the gray levels of the original image are redistributed so that the histogram of the transformed image will take the form of a uniform 
distribution across a specified range of gray levels. This range is usually the display range of the display device. The mapping is one-to-one; 
thus for each gray level of the original image, every pixel that appeared with that gray level will appear with a unique gray level in the 
transformed image. However, multiple gray levels from the original image can map to a single gray level tn the transformed image. 


For color images, histogram equalization is not a computationally simple process because of the requirement that the hue of each region 
remains the same before and after histogram equalization. To meet this constraint, the color image balanced histogram equalization algorithm 
calculates the equalization mapping for the intensity image, where the intensity image is obtained as the average of three pnmary images, the 
transformed primary images are calculated from the histogram-equalized intensity image. In this manner, the hue of each region of the image 
remains constant The algorithm operates as follows. The offsets of the color image intensities from the average intensity level are calculated, 
and the transformed color levels are calculated as the transformed intensity level plus the original offsets. For example, consider a single pixel. 
If the three original images' intensity values were red =» 140, green * 150, and blue » 1 10. then the average intensity at that point is I = 133. If 

the mapping derived by histogram equalization was 133 — >175, then the output color levels for that location are red — 182, green — 192, and 
blue = 152. 


Images transformed with this algorithm will exhibit full dynamic range, and the hues of the regions of the image will not change. This 
may be shown as follows. A three-channel color image can be equivalently represented by an HIS image, where HIS stands for hue-intensity- 
saturation. The hue image represents the color of the regions of the original color image, where the magnitude of the hue is proportional to the 
percentages of the three primary colors, red, green, and blue. The intensity image is simply the arithmetic average of the three color images. 
The saturation image represents the strength of the color. The range of the hue image is 0 to 359, the range of the intensity image is 0 to 255, 
and the range of the saturation image is 0 to 1. (The hue and intensity images can be archived as integer arrays, but because the range of the 
saturation image is 0 to 1, it is archived as a real-valued array.) 

For the HIS color space, the hue is calculated as a function of the ratio of linear functions of the three color image intensities. 
Specifically, this function is 


Hue 


l 

cos 


I((R-G) + (R-B)} 
7(RG) 2 + (R-BXG-B) 


where R. G. and B are the red, green, and blue intensities. If B>G, then the hue = 2x-hue. 

When R = G = B, the hue is undefined. 

Let the three original chromatic levels at each pixel be represented as R = 1 + A R , G = I + A q, and 
B = I + A B , where I is the intensity value of the pixel in the original image. Let the output color levels be R’ = 1’ + A R , G* = I' + A q, and 
B' a r + A B , where I’ is the intensity value of the pixel after transformation. Because R-G = R'-G\ R-B = R’-B', and G-B = G’-B', the 
magnitude of the hue is unchanged by the transformation. Therefore, the color information that was present in the original scene, but was not 
discemable because of the low dynamic range of the image, is preserved. This characteristic of the algorithm will guarantee that the transformed 
image is a good representation of the original scene because its colors are faithfully reproduced. 
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4 haaed approach f* «A>ptive Color Image Balanced Histogram Equalization would use a measure of an image's contrast in i local 

Exilic d^mLic range waskw (perhaps caused by shadowing) and apply the al^thm » that region. The algorithm 
obtaii^&om previous frames’ processing results to assist m the identification of shadows. 


-> i 5 Color inure Accentuation— Color image accentuation is a process whereby the image's sharpness is augmented by mcreasing Ac 
MtwaSfSf S^ge CoKSon may be increased as follows. The offseuof each of the original image mtensmes ftom the intensity 
image values ate calculated, and each of these offsets is amplified by a factor K, where K > 1. 


We can represent the quantities MAX and MIN (where MAX and MIN are the maximum and minimum of the three primary image 
intensities): 


MAX = I + Amax 
MIN = I + Amin 

The magnitudes of MAX and MIN, after transformation, are defined as 


MAX' = I + K Amax 
MIN’ = I - K A M1N 

The transformed image’s saturation is 

MAX’ - MIN 
S ■ : — 

MAX 

I-fKA MAX ~** KA M IN 

i + ka max 
a max +a min 

a 

K' + A MAX 


As K-* the term I/K in the denominator will decrease, thereby causing the saturation to increase. With die contracts that Amax is positive, 
Amin » positive, KA MIN £1, and KA MAX S255-I, the maximum value thal S’ can attain is 1, as K-» ~ It can beseen ^ the foUmrag 
argumentsthat the hwt of each region is unchanged. The color levels of the image after accenhanon can be represented as R= J 
v Kdc, and B’ - 1 + Kin. The magnitudes of each of the subtracted pairs-R’ - B’. R' - O’, and O’ - B’-are equal to K tunes the magnnude 
ofthe respective subtracted pair before accentuation. When these values are used tocalculatc the hue for aspecrfic pixelrf ^tff g^t he to 
K can be brought out of the numerator and the denominator, which means thal the magnitude of the hoe component will not change. 
Therefore, this algorithm also faithfully represents the hue of the original image. 

This technique may be employed to increase the saturation of colon of each of the regiooa of the image. The effect is to make small 
surface detail more distinguishable. 

or can be estimated fairly accurately. 

Constrained inverse filtering is a specific fonn of inverse Storing, It is a restoration rechmquelhal ™=rnpts * SI taSid l Z 

opticri transfer funcrion on an image. fibK^SSjdt^S^. 

restored image for a specific model of the image formation process, constrainea inverse * * ' . 5 ~| to nrevent the noise nreseni in 

with a constraint that thTnorm squared of the restored image is assmall as possiMe. 

the observed image from appearing at too great a level in the restored image. The model of the image formation process employes 
g (t, w) = h(t, w) * f(t, w) + n(t, w) 

where gft, w) is the observed image, hftw) is the PSF of the degradation function. Rt,w) is the original, ondistorted image, nftw) is 0- 
mean, white Guassian noise process, and x*y represents the two-dimensional convolution of x and y. 


The discrete representanon of the bgge is ^^^g g<hw^. • * <* jg*« * ^ 


ia- 

i) 


Ssnete'n^J^^of inu^e original intige, the PSF, and the additive noise, restively. The model 

equation for discretized images is 


K-t 

hr 2 , 

k«0 


L-l 


M> 


h U 1 i-k.j4 tj 


+ n. 
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If the dimensions of the spatial images, {gjj}, {fjj}, and (h^j), are the same, we can transform the previous equation into a frequency 
domain representation. Let N and M be the row dimension and column dimension, respectively, of the three images. These dimensions can 
be selected arbitrarily, but the values selected should be greater than J+L-l and I+K-lin the horizontal and vertical directions, respectively, to 
prevent the convolution window from extending off the image in the previous equation. They can also be selected as a power of 2, so that 
efficient implementations with the Fast Fourier Transform (FFT) are possible. 

If Fourier Transforms of the observed image {gjj}, the restored image {fjj}, and the PSF {hy} are defined as G(u,v), F(u,v), and 
H(u,v), respectively, then the frequency domain representation of the constrained inverse filter is: 


F(u,v) 


H(u,v)* 

H(u,v)* H(u,v) + y 


G(u,v) 


where H(u,v)* is the complex conjugate of H(u,v), and y is an arbitrary constant that controls the magnitude of the norm squared of the 
estimate of (fij). 

2.2 Tracking and Bandwidth Reduction --The tracking algorithm operates on a monochrome image to detect man-made objects in the field of 
view and track them over multiple images. The major elements of the tracking algorithm are multithreshold segmentation, boundary tracing, 
linearity filter, connected component analysis, and silhouette matching. Multithreshold segmentation is used to identify regions of the image 
with relatively constant intensity. A boundary tracing algorithm produces boundaries of regions that are passed through a linearity filter to 
determine which regions contain straight edges identifying them as part of a man-made object. Once the man-made pieces are assembled into 
objects by connected component analysis, tracking is performed by the silhouette matching algorithm, which compares silhouettes of objects in 
successive images to determine relative motion. Figure 1 is a data flow diagram of the tracking and bandwidth reduction algorithm. 

The following subsections describe the algorithm for each component function of the tracking algorithm. Due to the large degree of 
commonality between tracking and bandwidth reduction, the latter is discussed here in subsection 2.2.8. 

2.2.1 Window Average -The window average is a simple data-independent operation transforming the original intensity image into a smoothed 
intensity image. Its function is to help remove sensor noise in the image, ensuring that distinct regions in the image have relatively constant 
intensity. The operation computes a new value at each pixel position by averaging the input intensities of pixels in a window about the point. A 
5x5 window was found to work well on test imagery. 

2.2.2 Monochrome Segmentation -Monochrome segmentation divides the smoothed input image into different regions, where each region is 
characterized by a "nearly'’ uniform gray level. Labels are selected to represent different intensity ranges, and each region is appropriately 
labeled. This is accomplished in three major steps. First, an intensity histogram of the image is computed. Second, this histogram is searched 
for local minima and maxima, which define the intensity ranges corresponding to interesting regions in the image. Each distinct range is 
assigned a label and an upper and lower threshold defining the range. The third step is to apply the thresholds to the image, replacing the value 
at each pixel with the appropriate label. A list containing the labels and the average intensity values of their corresponding regions is also output 
for use by the bandwidth compression algorithm. 

2.2.3 Boundary Tracing -Boundary tracing is a data transformation algorithm whereby encoded region boundaries are obtained from a labeled 
image. The encoded region boundaries (silhouettes) arc much more compact than a complete image, requiring about four bits to store each 
image pixel that is on the boundary of a region. Conceptually, the algorithm interrogates each pixel of the image in an orderly fashion. At each 
point, the current pixel is examined to determine if it is on the boundary of a region that has not been (raced yet If so, the algorithm traces the 
boundary, ending at the same pixel where it began. The algorithm proceeds to examine the next point in search of more regions. When every 
image point has been examined, all of the regions have been traced. 

2.2.4 Linearity Filter --A linearity filter is applied to the set of silhouettes. This filter computes a measure of linearity for each silhouette to 
determine if they correspond to man-made objects. The output is a binary image with nonzero values at boundary points of regions whose 
silhouettes were relatively linear. 

The computation of the linearity filter is illustrated in Figure 2. Using a sliding window of width w, the angles si and s2 formed by the 
current point and the endpoints of the window are determined. The difference between these two angles is the curvature. For each silhouette, 
the linearity measure is the average curvature of all boundary points. If this measures exceeds a predetermined threshold, the boundary points 
of the region are set in the output binary image, indicating that the silhouette corresponds to a man-made object 

2.2.5 Connected Compon ent Analysis -Connected component analysis is a general-purpose function that identifies connected sets of pixels in 
an image. Each connected set is identified with a label in an output labeled image and a location in an output feature file. In this instance, 
connected component analysis operates on the binary image output from the linearity filter to determine which segmentation silhouettes (and 
therefore regions enclosed by these silhouettes) belong to the same object. Since different components of an object may have different 
intensities, an object may be segmented into several adjacent regions. Since the regions are adjacent, they can be grouped into one component 
by connected component analysis. 

2.2.6 Trace Components— This algorithm uses the starting locations and the labeled image output from connected component analysis to trace 
the boundary of each component (which in this case is a single target). Because the starting locations are known, this algorithm is considerably 
more efficient than the boundary tracing algorithm described in subsection 2.2.2. Thus, in this algorithm, it is not necessary that each image 
point be visited; however, the same method for tracing a region boundary is used. 

2-2.7 Fast Silhouette Matching- Fast silhouette matching compares the silhouettes of targets found in the current frame to stored silhouettes of 
targets tracked in previous frames. Depending on which new targets match which previous targets, the tracked target information is updated, 
and any new targets are added to the track list. To match new and previous targets, each new silhouette must be compared to each old 
silhouette. The match scores are used to determine which new and previous targets correspond. 
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To compare one new silhouette with one previous silhouette, the <x y) translation is ^ 

coincident boundary points of the two silhouettes. In theory, this is done by considering each possible (*.?>*»“ *uon ; ™ JWJJJ 
nnmS Of coincS boundary points. In practice, each point of the previous silhouette is considered as the point that uleally matches tiiefirst 
noint of the target silhouette Then the two silhouettes are traversed simultaneously, incrementing counters m a two-dimensional histograrr u 
ES °irrc e o=:o Kl tnu.sla.ioa nttcassiuy to make .he new and previous boundary points On 

completion, the histogram is searched for the maximum. The maximum value is compared to the length of the silhouettes to determine 

accuracy of the match. Essentially this is a variation of die Hough Worm. Choosing txw«pondmg “ 

makinga hypothesis about the telauve motion between images. Incremenung a counter is equivalent to voting far a PMcuJar (xy) mono 
vector. The peaks correspond to the most likely motion vectors. The actual motion vector generally gamers the most votes. The advantage ot 
such a technique is its robustness and relative insensitivity to noise. 

This is a very simple operation that combines several intermediate tracking results to be transmittal to a remote 
locanon ior siorugc or vrewmg. The essential information from the image is the target signature, complete with as much detail ot the target 
; tself as possible. The region silhouettes produced by the boundary tracing algorithm arc processed by the linearity filter and marked inthcaung 
whether that silhouette passed the linearity filter or not. Then the bandwidth reduction operation can look at each region silhouette and select the 
ones that were considered part of a man-made object 


Knowledge-based techniques for image interpretation are more robust than conventional techniques because they can lden^ symbol c 
image features ofthe basis of incomplete or imprecise information obtained from the image. Classical techniques, ui ' £ 

specific features of objects from images on the basis of the degree of match between the actual features and a 

features. When the degree of mismatch is sufficiently great, as determined by composing the magnitude of a degree of match metric to a 
threshold, the algorithrmrejcct the conclusion that the feature was observed. However, knowledge-based techniques d ° 
the hypothesis - they associate a "confidence factor" with the hypothesis transfer the positive and negative evidence obtained to date for that 
features a database If evidence is found that either confirms or refutes the existence of the specific feature, the database can be revised. It is 
SSS5SS5 knowSge based systems that makes them invaluable for image interpretation in unstructured environments, such as the 
environment These systems have the capability of deriving conclusions from ^afuti^^ 
information and maintaining the history of deductive steps applied to reach those conclusions to permit opnmal utilization of all available 

information at any single instant of time. 

There are four generic categories of knowledge-based scene interpretation algorithms, which are: 

1. Scene Labeling 

2. Temporal Resolution 

3. Context-based Resolution 

4. Knowledge-based Feedback Control for Resegmentation 

Each of these techniques will be explained in succeeding paragraphs. 

1 1 Srf»n<* I ibdine— The reliability and accuracy of each of the image processing functions tabulated in Table 1 will be enhanced with an 
unj cnSnga. ii! scctccLL for the image being evaluated. The context is deduced from a set of labels applied .0 ihe scene by a scene 
labeling algorithm. 

Honeywell has developed a Reasoning Region Classifier (RRC) i2 to identify and test the knowledge pertinent to each of a specific class 
of regions RRC is a production rule system, with explanation facilities, whose goal is to characterize image sub-regions of interest, based on 
vision system observable featues, such as region uniformity, texture smoothness, topological features^ etc Thl * s y s jm 
implemented for the classification of man-made and natural objects in air-to-ground imagery, but it could be easily modified to discriminate 
obfects for the space scenario. The model of the scene structure is a hierarchical database, which has the label entire scene at the root, and is 
subdivided at each level of the hierarchy as the classification of scene objects becomes more specific. The search for the true classification of a 
specific region or object is performed on the subtree which has the highest confidence level based on the production rules. 

3.2 Temporal Resolution -Temporal Resolution is a technique for the resolution of conflicts that result from region classification for region 
labels. It consists of the following steps: 

1 Identify a sequence of frames which have been segmented into regions each of which desplay a region in the neighborhood of the 
candidate region R. 

2. Determine whether the classifier result on the candidate region, say R, in the present frame, is consistent with the classifier results on 
the portion of the image corresponding to this region in past frames of the sequence. 

*3. Otherwise, modify the classifier result R by multiframe decision smoothing. 

7 , 7 Context-Based Resolution-Con text- based resolution conflict removal combines region information and relational context information from 
^ current scene, for modifying classifier decisions that are inconsistent with the world model as represented in the hierarchical database. 

Conflict removal is performed by detecting inconsistent configurations in the scene. The production rules that are used by the context- 
based resolution technique are based on a priori world knowledge. 

3 4 Knowl edge-Based Feedback Control for Resegmentation -A scene is composed of multiple regions that have different sizes and shapes. A 
single segementation algorithm may not suffice to properly segment all these regions. Appropriate choices of segmentors based on the ancillary 
information are crucial Further, each algorithm nas an associated set of parameters. Proper setting of the values of these parameters has a 
major impact on the algorithm regardless of how robust it is. For example, a large window size in noise smoothing techniques can bltrc^e 
edges between two regions, thus, resulting in an erroneous region classifications. In this case, adaptive thresholding can remedy the problem. 
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There are two types of knowledge-based control methodologies that can perform equally well under variations of scene characteristics 
encountered in Space Station scenario. 

3.4.1 Open-Loop Control— The first type of control is called open-loop knowledge-based control. This is a process that directs the low-level 
processing through proper image operator selection and the associated parameter value selection. Open-loop control processing governs simple 
tiara reduction tasks such as noise removal and region of interest selection. 

The Open-Loop Control scheme derives the process goal from the information stored in Short-Term Memory (data obtained from the 
current image) and the knowledge base. The process goal is usually based on temporal and ancillary information such as previous frame 
processing results, the lighting conditions and a priori information for the radiometric and topological characteristics of the current scene. Rules 
in the knowledge base are used to derive the process goal. An example of process goal derivation is: 

IF (mission goal IS (satellite detection) IN (high clutter area)) 

THEN ((locate region with two parallel linear borders) AND (remove high frequency noise)) 

Based on the derived processing goal, the selection control identifies the proper image operators with their associated values. The 
selection is also generated by the rules stored in the knowledge-base. An example of the selection rules is: 

IF (remove high frequency noise) 

THEN RUN (window average routine) 

The process module then executes the selected image operators with the given parametric values. The output of the process is passed 
directly to the next low level process. The process module can also generate some knowledge, such as the image contrast, which can be used 
by other processing modules. This generated knowledge is fed into the knowledge-base for subsequent use. 

3.4.2 Feedback Control -The second type of control is called the feedback knowledge-based control process. It is designed for governing 
complex low-level processes such as segmentation and color image enhancement processes. 

Similar to the open-loop control process, feedback control derives the process goal from the knowledge in the knowledge-base and 
short-term memory, and selects the appropriate image operators and their parametric values. Then, the image operators arc applied to the image 
and the results arc passed to a process evaluation module. The process evaluation module determines the next processing step. The evaluation 
module either 1) accepts the output and passes it to the next processing module, 2) feeds the image back to the same process module, 
recommending different image operators or parameter values for more refined processing, or 3) rejects the results and bypasses the process 
module. 

The evaluation decision is also based on rules and information acquired from the scene stored in the knowledge-base. An example of a 
region evaluation rule is the following: 

IF (Goal Size = small) AND (Goal Shape = rectangular) 

AND (Region Size = large) AND (Region Shape = rectangular) 

THEN (Resegment region with a lower threshold) 

These knowledge-driven control processes make the best use of all available information about the scene. Each processing module can 
achieve the best possible performance in satisfying the processing goal. Therefore, a high performance scene analysis system can be developed 
by synergistically integrating the low level processing results. 

4.0 Architecture Analysis for Parallelized, Multi-Pr ocessor Implementation of Knowledge-Based Algorithms 

Previous sections of this paper have briefly touched on the key Space Station tasks which can benefit from knowledge-based vision 
processing. Details of specific Space Station vision functions and their implementation have also been discussed. In this section, we overview 
key architectural issues in developing a hardware architecture and software methodology for implementing these vision functions. 

Developing real-time architectures for imaging systems is acknowledged as a difficult problem in many respects and remains a highly 
active research area. The key issues include: how to attain necessary and sufficient performance; how to program and maintain real-time 
systems; whether to use homogeneous or heterogeneous hardware; how to integrate processors with the environment; and how to develop 
planned/evolutionary approaches based on standards. A general solution to these central issues docs not exist. Instead they must be revisited 
for each new application considered. 

Statements of hardware performance requirements and capabilities usually are given simply in terms of the millions of operations per 
second (MOPS) needed for a set of functions or available from a system. A more critical measure of system performance would look at: 
operations per second (OPS) as a function of algorithmic requirements; power requirements; physical size and weight; and cost In short; 

Performance Measure = OPS(algorithm) / Watt env* $ 

Because transportation costs and limited space and weight budgets are key drivers in Space Station construction, the key elements of this 
metric should be throughput as a function of algorithm performed and total volume required to achieve this throughput Weight and power are 
typically correlated to volume for a given technology, and the desire is always to minimize cost consistent with achieving functionality. 
Typically, computer architects design systems in an attempt to keep functional units (e.g., arithmetic logic units or multipliers) maximally busy 
because algorithmic performance requirements arc specified in terms of the number of adds, multiplies, etc. Applying this approach to image 
processing architectures leads to designs in which 90%+ efficiency is achieved but on only 2-5% of the total processor hardware. Maximizing 
the throughput-to- volume ratio leads to more compact systems in which functional units arc not necessarily fully utilized and is a logical 
approach for signal and image analysis architectures destined for the Space Station. 
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The tradeoff between using a heterogeneous or homogeneous processor architecture is a 
system. The tradeoff is driven by algorithmic requirements as well as issues of system expandability, programmability, and flexibility. Current 
robust algorithmic paradigms for imaging systems subdivide the processing steps into various categories. An image understanding paradigm 
which has been useful in developing computer architectures is shown in Figure 3. This paradigm categorizes algontlunic functions according to 
data structures and processing functions. It is straight-forward, practical, and robust to directly translate sucn paradigms to hardware systems 
as indicated in Figure 4. Such an approach leads by nature to a heterogeneous architecture and from expenence tends to minimize system 
volume. In general, more specialized hardware modules lead to a more compact system, but maximizing the throughput-to-volume ratio in this 
fashion must be balanced with expandability, programmability, and flexibility requirements in the Space Station application. 

Two aspects of programmability become issues for real-time image processors. First, image processing hardware mustb^igned to 
utilize a high degree of parallelism at all levels to achieve high performance. The software methodology tmd tcwl setmust provide adequate 
means to deal with parallelism and must bridge the gap between coarse grained high-level languages (e.g., ^ 

fme-grained machine languages (e.g., microcode). Any inefficiency m the translation or completion process directly impacts the total system 
hardware requirement. Second, a software methodology intended for use with heterogeneous architectures must support all processor typers m 
an integrated fashion. These arc especially important issues in the Space Station setting where software development and maintanence costs wil 
likely be the dominant portion of total imaging system cost. 

The application environment affects imaging system architecture in many important ways. In the Space Station environment, factors 
such as fault tolerance and recovery, reliability, and testability are clearly important to safe and effective use of any mission critical computing 
equipment. In addition to these more or less generic considerations, very specific design details can be influenced by 'the envirMment. For 
example, electrical and radiation induced noise effects of the space environment lead naturally to consideration of optical interconnect for high 
data rate sensor channels. It is also logical to consider performing sensor specific preprocessing functions local to the sensor to reduce or 
eliminate channel induced noise. A broader environmental issue is the type and number of video sensors which can be active simultaneously. 
An architecture is needed which can readily switch between sensors and sensor types. 

A final high-level issue with significance to the Space Station application is the ability of the selected architecture to ?dapt in ar. 
evolutionary fashion to evolving mission requirements. Tc achieve such an adaptation capability requires a so-called open architecture in 
which modules may be added or replaced. Designing an open but heterogeneous architecture is difficult in that each element of the architecture 
brings specialized interconnection, software, and other requirements. Maximum use of standards is a necessity to successfully deve.oping such 
an open architecture. 

A specific image processor implementation for Space Station applications has been developed and is reported in a companion paper [ 1 3]. 
This Video Image Processor (VIP) design is based on careful consideration of the broad issues discussed above and on me specific 
requirements of the image processing tasks and algorithms discussed in earlier sections of this paper. Over 150 architectural vananons were 
analyzed using advanced computer modelling techniques. The result, illustrated in Figure 5, is a two- level architecture us.ng special purpose 
high-performance image pixel processing hardware operating in a pipelined fashion combined with a distributed shared-memory 
multiprocessor. These two levels perform the image frame processing and combined region and symbolic processing functions from the 
taxonomy of Figure 3. The relatively low update rates specified for VIP allow the array processing and general purpose computing functions of 
Figure 4 to be combined in the multiprocessor. 

Although the VIP architecture satisfies the essential processing requirements for the knowledge-based vision algorithms previously 
described and provides essential growth room, numerous architectural research areas with direct application to the Space Station remain to be 
explored. These include: 

Sensor Preprocessing. Gallium arsenide technology provides the capability to integrate analog, digital, and optical interconnection 
circuitry monoluhically This capability may be used to advantage in Space Station sensor preprocessing by combining analog to digital 
conversion hardware, preprocessing logic (for noise suppression, detector compensation, and bandwidth compression), and nigh speed optical 
data channels on a single chip located at the sensor. 

Programming Methodology . Two research areas relevant to programming heterogeneous signal and image processing systems arc 
being explored by us. The first is a hardware array processor architecture designed to perform certain run-time resource management functions 
through special hardware constructs [14]. This approach can out-perform static (e.g., compile-time) resource allocation and leacs to a more 
productive throughput-to-volume ratio then software-based dynamic allocation schemes. The second approach is a normal form language, 
IMP, which provides the programmer with manageable access to hardware parallelism rather then attempting to hide parallelism. This 
approach gives the programmer a homogeneous software environment for programming a heterogeneous system - hardware modules may be 
readily added or modified within the context of IMP. 

Cellular Architectures . Image processing architectures based on collections of simple cellular processors [15] hold significant potential 
for maximizing the throughput-to-volume ratio in space-borne applications. A new pixel-processing architecture based on a parade 
recirculating pipeline (PREP) is under development by us. This architecture avoids the classical computnoon-I/O-memory balance problem to 
achieve high pixel- processing performance in an extensible and high-order language programmable fashion. 

Evolutionary Architectures . One research effort recendy completed by us involved definition of an integrated signal and image processing 
subsystem using hardware, software, and mechanical standards in an open architecture configuration. Our architecture researca laboratop' 
(ARL) combines a multiprocessor environment with special purpose hardware in just such a configuration and allows us to plan and rapidly 
execute new processor module and system development in an evolutionary fashion. 


For the Color Image Enhancement Algorithms, the performance of the three algorithms was evaluated by synthesinn^^^fion 
which might be encountered for real space imagery for the high-quality photographs and then quantitatively comparing the degraded ’ 

after enhancement with each of the three algorithms, to the original image. A block diagram of the quantitative evaiuanon ^edure ^s 
Figure 6. For the Tracking and Bandwidth Reduction algorithms, a sequence of 8 frames at 1 second internals were ^?m 

41-C "Video Highlights" video tape, where the imagerv depicts the SYNCOM satellite rotatng in space near the Space Shuttle, diortly a 
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deployment, against a cloud-covered earth background. The Tracking function's accuracy was empirically evaluated by comparing the 
algorithm's estimate for the two-dimensional change in location of the satellite (displacement vectors) to the best-guess at the actual displacement 
vectors, which were estimated by visual inspection of the gray levels of each successive pair of images. The maximum error in the estimated 
displacement vectors for the eight frame sequence was 1 pixel vertically and horizontaly. 

Each of the Color Image Enhancement algorithms were evaluated for the task of restoring degraded imagery for a range of image 
degradations in order to accurately characterize the algorithm's performance in terms of an empirically derived model for its behavior. Tbe 
Color Image Balanced Histogram Equalization algorithm and the Color Image Accentuation algorithms are efficient computational techniques for 
die restoration of dynamic range for color imagery. Here the dynamic range is the difference between the maximum intensity value and tbe 
minimum intensity value of the luminance image. A set of three degraded color images were generated, one which had a dynamic range equal to 
50% of the original image's dynamic range, one with 62% dynamic range and one with 85% dynamic range. These images were then restored 
with the Color Image Balanced Histogram Equalization algorithm and the Color Image Accentuation algorithm, in that order. Mean-squaie- 
error masures were then employed to quantitatively evaluate the accuracy of the restorations. These mean-square-error measures were the 
output luminance image signal-to-noise ratio and the output chromatic signal-to-noise ratios. For ail cases but one, the quantitative measures 
demonstrated that the Color Image Enhancement algorithms did restore full dynamic range to the test imagery and also did not distort the 
intensity or chromatic information of the images. Further details can be found in 2. 

A few of the results of the experiments with the Color Image Histogram Equalization and the Color Image Accentuation algorithm are 
presented in Figures 7 through 10. Figure 7 is the original image. Figure 8 is the degraded image with a 50% reduction of dynamic range with 
respect to the original image. Figure 9 is the result obtained by processing the image with the Color Image Balanced Histogram Equalization 
algorithm and Figure 10 is the result of increasing the image's saturation after histogram equalization. Inspection of these images demonstrates 
that full dynamic range has been restored. 

6.0 fomdusispa 

There are a multitude of applications where knowledge-based techniques may be employed to improve the performance of image 
interpretation algorithms, for space applications. Because the benefits of knowledge-based image interpretation algorithms; increased algorithm 
reliability and increased robustness, arc of great importance in the unique space environment, it is apparent that any future architectural concept 
development efforts should take knowledge-based techniques into consideration. 
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Table 1. Cross reference between applications and algorithms. 
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Figure 2. Computation of the Linearity Filter. 


INTENSITY I MAC! 


INTENSITY 

IMAGE 


LOW-PASS FILTER 


HISTOGRAM 

CALCULATION 


INTENSITY IMAGE 


THfICSHOlD 

| INTENSITY VALUES 

DETERMINATION 

1 



A# .*irnfm««r 

i 




\n * un*9* rt***nj 

P - Mcmmt * • • 

MUinTHRISHOlO 

IMAOE 

i REGION J 
SILHOUETTED^] 

•andwidth 

REDUCTION 

L ^ 

\ fp • fiMtiiiNiAi ^mtoer 

! A • 9*v*«u« PrWMMf 

1*1 - ***** *♦*« 

I tn - tyMMHC WKG — 


REGION 

SILHOUETTES 




« m n i* >l ptwi* 




Figure 5. Video Image Processor (VIP) Conceptual Architecture. 
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Figure l. Functional decomposition of the Tracking and Bandwidth 
Reduction algorithms. 


Figure 6. Color image enhancement algorithm performance is evaluated 
with various error measures. 


Figure 7. Original image. 
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Figure 3. A Taxonomy of Image Understanding Operations. 


Figure 8. Degraded image: 50% dynamic range. 



• SP = signal processor 

• AP = array processor 

• GPC~ general purpose computer 

Figure 4. A Hardware Architecture Based on the Taxonomy of Figure 1. 



Figure 10. Restored and enhanced image. 
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